This invention relates to the field of digital image processing method and apparatus technologies. More particularly, the invention relates to image processing methods and image processing apparatus which can apply this image processing methods for typical use with digital photoprinters that read film images photoelectrically to produce prints (photographs) reproducing the images and which are capable of achieving one of the following results: producing a high-quality image even if the input image is taken with low-performance lenses as in films with lens, inexpensive compact cameras and low-cost digital cameras; correcting aberrations such as chromatic aberration of magnification and distortion that develop in the images taken with those low-performance lenses; producing a high-quality image free from the image deterioration due to lens aberrations even if the input image is taken with those low-performance lenses; and particularly performing image processing on the image data representing the image recorded on an image recording medium.
Heretofore, the images recorded on photographic films such as negatives and reversals (which are hereunder referred to simply as “films”) have been commonly printed on light-sensitive materials (photographic paper) by means of direct (analog) exposure in which the film image is optically projected onto the light-sensitive material to achieve its areal exposure.
A printer which relies upon digital exposure has recently been commercialized as a digital photoprinter, in which the image recorded on a film is read photoelectrically, converted to digital signals and subjected to various image processing operations to produce image data for recording purposes; recording light that has been modulated in accordance with the image data is used to scan and expose a light-sensitive material to record a latent image, which is subsequently developed to produce a finished print (photograph).
In the digital photoprinter, images are handled as digital image data and the exposing conditions for printing can be determined by image (data) processing. Hence, processing operations such as the correction of washed-out highlights or flat (dull) shadows due to the taking of pictures with rear light or an electronic flash and sharpness enhancement (sometimes referred to simply as “sharpening”) can be effectively performed to produce prints of the high quality that has not been attainable by the conventional direct exposure technique. In addition, the synthesizing of images and characters can be accomplished by image (data) processing and, as a result, prints can be output after editing and/or processing operations have been performed freely in accordance with specific uses.
Aside from the images recorded on films, the digital photoprinter can also output prints of images recorded with digital cameras or processed with computers. Other than being output as prints, images can be supplied to computers and so forth or stored in recording media such as floppy disks; hence, the image data can be put to various non-photographic uses.
Having these features, the digital photoprinter is basically composed of the following units: a scanner (image reading apparatus) that illuminates the film with reading light and captures the projected light to read the image on the film photoelectrically; an image processing apparatus that performs specified image processing on the image data captured with the scanner or the image data supplied from a digital camera or the like, thereby producing image data for image recording and exposing conditions; a printer (image recording apparatus) that scans and exposures a light-sensitive material to record a latent image on it, for example, by scanning with optical beams in accordance with the image data output from the image processing apparatus; and a processor (developing apparatus) that performs development processing on the printer exposed light-sensitive material to produce a print reproducing the input image.
Users in general who intend to take ordinary pictures seldom use expensive, high-performance cameras such as a single-lens reflex camera but they normally use so-called “compact cameras” that are inexpensive and which are capable of automatic exposing and focusing. Most recently, there are a lot of users who prefer using so-called “films with lens” by the reason of easy handling.
In cameras such as a single-lens reflex camera that needs a cost to some extent, high-precision lens elements are used, and a plurality of lens elements are combined to record images of very high quality.
In contrast, films with lens and inexpensive compact cameras cannot afford the use of costly lenses and only one or two lens elements may be adopted. With such lens design, images of adequate quality cannot be taken and the image reproduced on prints does not necessarily have high quality.
If the image recorded on films is deteriorated in quality, there is a case that the quality of the output image on prints cannot be adequately improved by the aforementioned corrections. Major causes of the deterioration of the image reproduced from films to be output on prints are lens aberrations such as “chromatic aberration of magnification” and “distortion” that originate from the low performance of the lenses mounted in the camera used to take the input image.
Color images are formed of three primary colors, for example, red (R), green (G) and blue (B). The refractive index (imaging magnification) of a lens, even if it is a single element, varies subtly with wavelength and differing refractive indices occur with R, G and B lights. In other words, even though the same position in a particular scene, a focused position on a film are slipped off and differ among the R, G and B lights. This is the phenomenon generally called “chromatic aberration of magnification” and the image reproduced from the film has a definite color divergence.
In order to obtain a satisfactory and appropriately recorded image, a plane of a scene of interest that is perpendicular to the optical axis must be focused on the same plane as the imaging plane perpendicular to the optical axis. In fact however, ordinary lenses have the imaging plane displaced along the optical axis and the resulting displacement of the focused position in the axial direction causes a distortion of the focused object. As a natural consequence, the reproduction of the image on the film is distorted.
Other causes of the image deterioration are the reduction of the brightness at the edge of image field which means a phenomenon in which the peripheral area of the image looks darker than the central area which is closer to the optical axis corresponding to the performance of the lens used, and the point spread function (PSF) which is attributable to differing focal positions in the plane of the film.
As noted above, if one uses a camera such as a single-lens reflex camera that needs a cost to some extent, high-precision lens elements may be used and a plurality of lens elements combined to correct various aberrations including chromatic aberration of magnification, distortion, deterioration of marginal lumination and PSF and an appropriate image can be recorded on the film.
However, cameras such as films with lens and compact cameras required to be a low cost can not use high-cost lenses and aberrations will develop in the images recorded on films. As a result, the images reproduced on prints will eventually have color divergence and distortion.
To deal with this problem of image deterioration involving the difficulty in improving the quality of output images on prints, techniques have been proposed in connection with image processing methods and apparatus that correct image aberrations in accordance with the characteristics of lens aberrations that are obtained via certain image acquisition means and two typical examples of such technology are disclosed in Unexamined Published Japanese Patent Application (kokai) Nos. 311425/1994 and 281613/1997, the latter being assigned to the present Applicant. According to these patents, the proposed technology can correct aberrations due to lenses and prevent the deterioration of image quality in the marginal area, thereby ensuring the production of high-quality images at all times.
Specifically, Unexamined Published Japanese Patent Application (kokai) No. 281613/1997 proposes a process of correcting the problem of deterioration of marginal lumination in a photographic processing apparatus and method. In the process, the quantity of light f(i,j) on an image in a given pixel position (i,j) is multiplied by a correction coefficient g(i,j) based on lens characteristics and the obtained product f′ (i,j) is substituted as the corrected quantity of light in the pixel position (i,j). To make the correction for the entire part of the image, j is first moved with i held constant and then i is moved, or alternatively, i is first moved with j held constant and then j is moved; in either way, the whole image can be corrected. When correcting distortion and chromatic aberration of magnification by this method, the position of the subject in an image of interest is changed so that a huge frame memory is required to store the information about all pixel positions of not only before correction but also after the correction. In addition, a circuit is necessary that performs two processing schemes for i and j as described above and this not only increases the cost of the apparatus but also causes a significant drop in the correction speed, thus there is a problem the practical use of the apparatus is difficult.
On the other hand, unexamined Published Japanese Patent Application (kokai) No. 311425/1994 discloses an image correcting apparatus capable of rapid image correction in accordance with the characteristics of lens aberrations. In this apparatus, the subjects of the correction are the amount of defocusing, the decrease in the quantity of light, and the degree of unsharpness in the hue and chromas of a color image. According to the disclosure, a quantity of deterioration in each of these correction subjects increases as it goes from the center of the image to the peripheral area so that the data of specified patterns that are increased as going from the center toward the peripheral in an image area that is formed by concentric circles or squares extending radially outward from the center of the image are only used for each of given lens characteristics as correction enhancement coefficients for correcting these correction subjects. This approach is capable of rough correction but not image corrections aberrant from the patterns that is prepared beforehand. Hence, there is a problem that it is impossible to perform appropriate corrections according to the characteristics of individual taking lenses.
If one wants to accomplish the appropriate correction by this technology, correction enhancement coefficients must be made available for all patterns that are predicted for the given lens characteristics and to meet this need, a memory of sufficiently large capacity is required. What is more, if the available patterns are not simple concentric circles or squares whose center coincides with that of the image, an increased amount of data has to be calculated to correct matrices and the overall image processing speed is lowered. These practical problems with cost and processing speed are particularly serious in the case of print services that involve the volume reproduction of images.
As mentioned hereinabove, an image processing system is conventionally known that is able to perform various image processing schemes on the image data obtained by reading the film image recorded on photographic films or on the image data input from a digital camera or the like and which then outputs an image in various output modes such as by recording the image on recording materials such as photographic paper or storing the image data in information recording media. Compared to a conventional photographic processing system that records the film image by areal exposure, the image processing system just described above can control the output image quality by image processing onto the image data, thereby output images of high quality are realized.
Speaking of films with lens, the lens is usually composed of an inexpensive plastic lens that inevitably suffers from great amounts of aberrations such as distortion and chromatic aberration of magnification. Hence, the film image recorded on a photographic film by exposure with the film with lens has a comparatively high level of geometric distortion according to the lens distortion (so-called “pincushion-type distortions) as typically shown in FIG. 25A (FIGS. 25A and 25B illustrate how an image consisting of multiple lines in a grid pattern appears if it is recorded on a photographic film by shooting with a film with lens); at the same time, a color divergence occurs at a comparatively high level due to chromatic aberration of magnification. To deal with this problem, distortion correction for correcting the geometric distortion of an image due to the distortion of the lens on the film with lens and magnification chromatic aberration correction for correcting the color divergence in an image due to the chromatic aberration of magnification of the same lens are being review in order to ensure that the image processing system described above can produce an output image of high quality from the original (input) image recorded with the film with lens.
With a view to increasing the speed of the various processing schemes to be performed with the above-mentioned image processing system, the contents of image processing and the value of a parameter to be applied in a specific image processing scheme have been set identical in specified directions, such as vertical and horizontal, of the image represented by the image data. For instance, if the image processing to be done is electronic scaling, the electronic scaling ratio is set identical in both the vertical and horizontal directions of the image represented by the image data to be processed.
What is unique about image data is that its volume is tremendous and that it represents an image having a two-dimensional extent. Hence, image processing schemes such as ones for correcting distortion and chromatic aberration of magnification are so much complicated in contents that they not only take prolonged time but also require storage means of large capacity. As a result, the image processing section capable of performing image processing schemes such as the correction of distortion and chromatic aberration of magnification is considerably complicated in configuration and, what is more, the processing performance of the image processing system is eventually deteriorated.
Consider, for example, the correction of distortion. First, distortion correcting data representing the direction and the amount of the shift in the position of each of the pixels in the original film image due to the distortion of a lens, as referenced to the inherent position of each pixel (the position of its grid point) composing the film image, are preliminarily measured and stored for each of the lens types used; given the image data to be processed, the distortion correcting data associated with the lens type used in actual shooting are captured; on the basis of the captured distortion correcting data, the positions of the pixels represented by their data in the case where no distortion is present are evaluated; and the density value at the inherent position of a particular pixel (the position of its grid point) is determined by arithmetic interpolation. Among the steps described above, arithmetic interpolation of the density values at grid points requires arithmetic estimation of the density value in the position of a particular grid point from the density values of a plurality of pixels surrounding said grid point (i.e., the pixels within a region having a two-dimensional extent, with the grid point lying in the center) should be repeated for the two-dimensional distribution of the multiple grid points that are to be processed. Obviously, this involves quite complicated procedures.
As is clear from the above, the distortion correction involves the shift of the pixel positions represented by the yet to be corrected image data and, hence, the shape of the outer edge of the image represented by the as-corrected image data also changes from a rectangular to a non-rectangular form (such as a barrel or pincushion shape) as the result of the aforementioned aberration (distortion) correction. Consider, for example, the case of correcting an image which, due to distortion, has suffered from a “pincushion-type” geometric distortion as shown in FIG. 25A. After the correction, the figure of the outer edge of the image represented by the corrected image data is “barrel-shaped” as shown in FIG. 25B. On the other hand, images generally have rectangular outer edges. Therefore, if the distortion corrected image data is simply output, blanks or regions having indeterminate density values occur in some areas of the output image (blanks or so-called “pixel holes” that appear in areas near the four corners of the image shown in FIG. 25B). The same defect occurs in the correction of chromatic aberration of magnification since it also involves the shift of pixel positions although it is very small.
As already mentioned, the content of image processing and the value of a parameter to be applied to a specific image processing scheme in the conventional image processing system have been set identical in specified directions, such as vertical and horizontal, of the image represented by image data. Hence, it has been impossible to perform different image processing schemes in different directions and this has reduced the latitude in the overall image processing operation.
Take, for example, the aforementioned case of performing electronic scaling with the conventional image processing system, in which the electronic scaling ratio is set to be identical in both the vertical and horizontal directions. In this situation, it has been impossible to perform a special image processing scheme such as a so-called “finish to slender” technique by which a human subject in the original image is finished to appear slender on the principal image.